Image processing apparatus, method, and storage medium capable of generating wide angle image

ABSTRACT

A digital camera  1  includes an image composition unit  52  and a composite position determining unit  53.  The image composition unit  52  acquires a plurality of images which have been captured continuously. The composite position determining unit  53  sets candidate areas for determining composite positions in adjacent images, from the plurality of images acquired by the composite position determining unit  52.  Furthermore, the composite position determining unit  53  determines composite positions of the adjacent images based on a degree of similarity between the set candidate areas. The image composition unit  52  combines the adjacent images at the determined composite positions.

This application is based upon and claims the benefit of priority fromthe prior Japanese Patent Application No. 2010-221329, filed Sep. 30,2010, and the entire contents of which are incorporated herein byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, method,and storage medium, and more particularly to an image processingapparatus, method, and storage medium capable of generating a wide angleimage.

2. Related Art

With regard to digital cameras, cell phones having an image capturingfunction, and the like, the angle of view for image capturing depends onthe hardware specification of the device main body, such as the focallength of the lens, the size of the image sensor, and the like.

Japanese Patent Application Publication No. 1999-282100 discloses atechnique such that a plurality of images are continuously capturedwhile moving an image capturing apparatus in a predetermined directionand are combined to generate a wide angle image, in a case of capturinga wide angle image, such as a panoramic image that is beyond thehardware specification.

Japanese Patent Application Publication No. 1999-282100 discloses atechnique that detects a characteristic point of a captured image ateach time of a plurality of times of image capture processing,horizontally combines image data of a plurality of captured images sothat the characteristic points of two adjacent captured images matcheach other, and thereby generates image data of a panoramic image.

SUMMARY OF THE INVENTION

It is an object of the present invention to generate a natural wideangle image.

In order to attain the object of the present invention, in accordancewith a first aspect of the present invention, there is provided an imageprocessing apparatus, comprising:

an acquiring unit that acquires a plurality of images which have beencaptured continuously;

a setting unit that sets candidate areas for determining compositepositions in adjacent images, from the plurality of images acquired bythe acquiring unit;

a composite position determining unit that determines compositepositions of the adjacent images based on a degree of similarity betweenthe candidate areas set by the setting unit; and

a composite image generating unit that combines the adjacent images atthe composite positions determined by the composite position determiningunit.

In order to attain the object of the present invention, in accordancewith a second aspect of the present invention, there is provided animage processing method of an image processing apparatus having anacquiring unit that acquires a plurality of images which have beencaptured continuously; the method comprising the steps of:

a setting step of setting candidate areas for determining compositepositions in adjacent images, from the plurality of images acquired bythe acquiring unit;

a composite position determining step of determining composite positionsof the adjacent images based on a degree of similarity between thecandidate areas set in the setting step; and

a composite image generating step of combining the adjacent images atthe composite positions determined by the composite position determiningstep.

In order to attain the object of the present invention, in accordancewith a third aspect of the present invention, there is provided astorage medium readable by a computer that controls an image processingapparatus having an acquiring unit that acquires a plurality of imageswhich have been captured continuously, the storage medium having storedtherein a program causing the computer to function as:

a setting unit that sets candidate areas for determining compositepositions in adjacent images, from the plurality of images acquired bythe acquiring unit;

a composite position determining unit that determines compositepositions of the adjacent images based on a degree of similarity betweenthe candidate areas set by the setting unit; and

a composite image generating unit that combines the adjacent images atthe composite positions determined by the composite position determiningunit.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a hardware configuration of a digitalcamera as one embodiment of an image processing apparatus according tothe present invention;

FIG. 2 is a functional block diagram showing a functional configurationfor the digital camera shown in FIG. 1 to carry out image captureprocessing;

FIGS. 3A and 3B are diagrams respectively illustrating image captureoperations in cases in which a normal image capture mode and a panoramicimage capture mode are respectively selected as an operation mode of thedigital camera shown in FIG. 2;

FIG. 4 is a diagram showing one example of panoramic image generated inthe panoramic image capture mode shown in FIG. 3B;

FIGS. 5A, 5B, and 5C are diagrams respectively illustrating a method ofdetermining composite positions when combining images for the purpose ofgenerating image data of a panoramic image;

FIG. 6 is a flowchart showing one example of flow of image captureprocessing carried out by the digital camera shown in FIG. 2; and

FIG. 7 is a flowchart showing a detailed flow of panoramic image captureprocessing from the image capture processing of FIG. 6.

DETAILED DESCRIPTION OF THE INVENTION

The following describes an embodiment of the present invention withreference to the drawings.

FIG. 1 is a block diagram showing a hardware configuration of thedigital camera 1 as one embodiment of the image processing apparatusaccording to the present invention.

The digital camera 1 is provided with a CPU (Central Processing Unit)11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus14, an optical system 15, an image capturing unit 16, an imageprocessing unit 17, a storing unit 18, a display unit 19, an operationunit 20, a communication unit 21, an angular velocity sensor 22, and adrive 23.

The CPU 11 executes various processes according to programs that arestored in the ROM 12 or programs that are loaded from the storing unit18 to the RAM 13.

The RAM 12 also stores data and the like, necessary for the CPU 11 toexecute the various processes, as appropriate.

For example, according to the present embodiment, programs forimplementing functions of an image capture control unit 51, an imagecomposition unit 52, and a composite position determining unit 53 shownin FIG. 2, which will be described later, are stored in the ROM 12 orthe storing unit 18. Therefore, each of the functions of the imagecapture control unit 51, the image composition unit 52, and thecomposite position determining unit 53 shown in FIG. 2, which will bedescribed later, can be realized by the CPU 11 executing processesaccording to these programs.

Incidentally, it is possible to transfer at least a part of eachfunction of the image capture control unit 51, the image compositionunit 52, and the composite position determining unit 53 shown in FIG. 2,which will be described later, to the image processing unit 17.

The CPU 11, the ROM 12, and the RAM 13 are connected to one another viathe bus 14. The bus 14 is also connected with the optical system 15, theimage capturing unit 16, the image processing unit 17, the storing unit18, the display unit 19, the operation unit 20, the communication unit21, the angular velocity sensor 22, and the drive 23.

The optical system 15 is configured by a light condensing lens such as afocus lens, a zoom lens, and the like, for example, to photograph asubject. The focus lens is a lens for forming an image of a subject onthe light receiving surface of the image sensor of the image capturingunit 16. The zoom lens is a lens for freely changing a focal pointwithin a predetermined range. The optical system 15 also includes aperipheral device to adjust focus, exposure, and the like, as necessary.

The image capturing unit 16 is configured by an optoelectronicconversion device, an AFE (Analog Front End), and the like. Theoptoelectronic conversion device is configured by a CCD (Charge CoupledDevice) type or a CMOS (Complementary Metal Oxide Semiconductor) typeoptoelectronic conversion device, for example. The optoelectronicconversion device optoelectronically converts (i.e., captures), at apredetermined interval, a light signal of an image of a subject, whichhas been incident and accumulated during the interval, and sequentiallysupplies the resultant analog signal to the AFE.

The AFE executes various kinds of signal processing such as A/D(Analog/Digital) conversion on the analog signal and outputs theresultant digital signal as an output signal from the image capturingunit 16.

Hereinafter, the output signal from the image capturing unit 16 isreferred to as “image data of a captured image”. Thus, image data of acaptured image is outputted from the image capturing unit 16 andprovided as appropriate to the image processing unit 17 and the like.

The image processing unit 17 is configured by a DSP (Digital SignalProcessor), a VRAM (Video Random Access Memory), and the like.

The image processing unit 17 collaborates with the CPU 11 to executeimage processing such as noise reduction, white balance, andanti-shaking on image data of a captured image inputted from the imagecapturing unit 16.

Hereinafter, unless otherwise noted, image data of each captured imageinputted from the image capturing unit 16 at a predetermined interval ora processing result thereof is referred to as “image data of a frame”,and each captured image expressed by the “image data of a frame” isreferred to as “frame”. In the present embodiment, such image data of aframe is employed as a unit of processing.

The storing unit 18 is configured by a DRAM (Dynamic Random AccessMemory) or the like, and temporarily stores image data of a frameoutputted from the image processing unit 17, image data of a panoramicimage in progress, which will be described later, and the like. Also,the storing unit 18 stores various kinds of data necessary for variouskinds of image processing.

The display unit 19 is configured as a flat display panel, formed, forexample, from an LCD (Liquid Crystal Device), an LCD driving unit, andthe like. The display unit 19 displays an image represented by imagedata provided from the storing unit 18 or the like, e.g., a live-viewimage that will be described later, as a frame unit.

The operation unit 20 includes a plurality of switches, as well as ashutter switch 41, such as a power switch, an image capture mode switch,and a playback switch, which are not illustrated. When one of theplurality of switches is pressed and operated, the operation unit 20provides to the CPU 11 an instruction assigned to the switch.

The communication unit 21 controls communication with other devices (notshown) via a network including the Internet.

The angular velocity sensor 22 includes a gyro or the like, detects avalue of angular displacement of the digital camera 1, and provides tothe CPU 11 the digital signal (hereinafter, referred to simply as“angular displacement”) indicative of the detection result. Here, theangular velocity sensor 22 is assumed to implement a function of anorientation sensor, as necessary.

Removable media 31 such as a magnetic disk, an optical disk, amagneto-optical disk, or a semiconductor memory is mounted to the drive23 as appropriate. Also, programs read from the removable media 31 areinstalled in the storing unit 18 as necessary. Furthermore, similar tothe storing unit 18, the removable media 31 can store various kinds ofdata such as image data and the like, stored in the storing unit 18.

FIG. 2 is a functional block diagram showing a functional configurationto carry out a series of processing (hereinafter, referred to as “imagecapture processing”) by the digital camera 1 shown in FIG. 1, fromcapturing an image of a subject up to storing captured image data of thesubject in the removable media 31.

As shown in FIG. 2, the CPU 11 is provided with an image capture controlunit 51, an image composition unit 52, and a composite positiondetermining unit 53.

It is to be noted that, as described above, the functions of the imagecapture control unit 51, the image composition unit 52, and thecomposite position determining unit 53 need not necessarily be installedin the CPU 11 as in the present embodiment, and it is possible totransfer at least a part thereof to the image processing unit 17.

The image capture control unit 51 controls the overall execution of theimage capture processing. For example, the image capture control unit 51can selectively switch an operation mode of the digital camera 1 betweena normal image capture mode and a panoramic image capture mode and canexecute processing according to the operation mode to which switchinghas been made.

In the panoramic image capture mode, the image composition unit 52 andthe composite position determining unit 53 operate under the control ofthe image capture control unit 51.

In the following, for ease of understanding of the image capture controlunit 51, the image composition unit 52, and the composite positiondetermining unit 53, before describing the functional configurationsthereof, a detailed description will be given concerning the panoramicimage capture mode with reference to FIGS. 3 and 4 as appropriate.

FIGS. 3A and 3B are diagrams respectively illustrating image captureoperations in cases in which a normal image capture mode and a panoramicimage capture mode are respectively selected as an operation mode of thedigital camera 1 shown in FIG. 2.

More specifically, FIG. 3A is a diagram illustrating the image captureoperation in the normal image capture mode. FIG. 3B is a diagramillustrating the image capture operation in the panoramic image capturemode.

In each of FIGS. 3A and 3B, the picture in the back of the digitalcamera 1 shows a real world view including a subject of the digitalcamera 1. The vertical dotted lines shown in FIG. 3B indicate positionsa, b, and c in the moving direction of the digital camera 1. Here, themoving direction of the digital camera 1 is intended to mean a directionin which an optical axis of the digital camera 1 moves when the userpivots about his/her own body to change the shooting direction (angle)of the digital camera 1.

The normal image capture mode is intended to mean an operation mode inwhich an image of a size (resolution) corresponding to the angle of viewof the digital camera 1 is captured.

In the normal image capture mode, as shown in FIG. 3A, the user pressesthe shutter switch 41 of the operation unit 20 all the way down whileholding the digital camera 1. Hereinafter, such an operation of pressingthe shutter switch 41 all the way down is referred to as “full pressoperation” or simply “full press”.

The image capture control unit 51 controls execution of a series ofprocesses up to a process of storing in the removable media 31, as arecording target, image data of a frame outputted from the imageprocessing unit 17 immediately after the user's full press operation.

Hereinafter, such a series of processes carried out under control of theimage capture control unit 51 in the normal image capture mode isreferred to as “normal image capture processing”.

On the other hand, the panoramic image capture mode is intended to meanan operation mode in which a panoramic image is captured.

In the panoramic image capture mode, as shown in FIG. 3B, the user movesthe digital camera 1 in a direction of black arrows shown therein, whilecontinuing the full press operation of the shutter switch 41.

While the full press operation is continued, the image capture controlunit 51 controls the image composition unit 52 and the like to repeattemporarily storing in the storing unit 18, image data of a frameoutputted from the image processing unit 17 immediately after each timean angular displacement cumulatively provided from the angular velocitysensor 22 reaches a constant value.

After that, the user gives an instruction for termination of thepanoramic image capturing by an operation of releasing the full pressoperation (hereinafter, referred to as “release operation”), i.e.,moving a finger or the like, away from the shutter switch 41.

When instructed to terminate the panoramic image capturing, the imagecapture control unit 51 generates image data of a panoramic image byhorizontally combining the image data of the plurality of frames thusfar stored in the storing unit 18 in the stored order.

The image capture control unit 51 then controls the image compositionunit 52 and the like to store the image data of the panoramic image inthe removable media 31 as a recording target.

Thus, in the panoramic image capture mode, the image capture controlunit 51 controls the image composition unit 52 and the like and controlsa series of processing, from generating image data of a panoramic imageup to storing it in the removable media 31 as a recording target.

Hereinafter, such a series of processing in the panoramic image capturemode carried out under control of the image capture control unit 51 isreferred to as “panoramic image capture processing”.

More specifically, under control of the image capture control unit 51,the image composition unit 52 carries out processing as follows:

Each time the digital camera 1 moves by a constant amount (each time thecumulative value of the angular displacement thereof reaches a constantvalue), the image composition unit 52 receives an acquisitioninstruction issued from the image capture control unit 51, and acquiresimage data of one frame from the image processing unit 17. The imagecomposition unit 52 horizontally combines the image data of theplurality of frames thus far acquired in the acquired order, and therebygenerates image data of a panoramic image.

FIG. 4 shows one example of a panoramic image generated by the imagecomposition unit 52 in the panoramic image capture mode shown in FIG. 3.

In the panoramic image capture mode, when an image capture operation asshown in FIG. 3B is performed, under control of the image capturecontrol unit 51, image data of a panoramic image P3 shown in FIG. 4 isgenerated by the image composition unit 52, and stored in the removablemedia 31.

The composite position determining unit 53 determines a compositeposition in a unit of a candidate area for the image data of each of theplurality of frames to be combined by the image composition unit 52.

Here, the candidate area is intended to mean a set of pixelsconstituting a line or a rectangle from among constituent pixels of theimage data of a frame. Hereinafter, the direction of the line or thelongitudinal direction of the rectangle is referred to as “lengthdirection”, the length of the line or the rectangle in the lengthdirection is referred to as “line length”, the direction orthogonal tothe length direction is referred to as “width direction”, and the lengthof the line or the rectangle in the width direction is referred to as“line width”.

Here, the line is intended to mean a set of pixels having one pixel inthe width direction and L (L being an integer larger than one) pixels inthe length direction. Assuming that the unit for line length and linewidth is pixel, a line is a set of pixels whose line length is L andwhose line width is 1. On the other hand, a rectangle is a set of pixelswhose line length is L and whose line width is W (W being an integerlarger than one and smaller than L). In short, if W indicating the linewidth can include 1, what is referred to as a “line” is a set of pixelssuch that the line length is L and the line width is W.

Although the length direction of a candidate area is not limited and canbe arbitrary, it is preferable that the length direction be a directionperpendicular to the direction in which the plurality of frames arecombined. It is because, in that case, the length direction of acandidate area coincides with the direction of the boundary line ofadjacent frames to be combined. In the present embodiment, since theplurality of frames are combined in a horizontal (landscape) direction,the length direction of a candidate area is assumed to be vertical.Hereinafter, as far as the present embodiment is concerned and unlessotherwise noted, it is assumed that the candidate area means a candidatearea in a vertical direction.

FIG. 5 is a diagram illustrating a method of determining a compositeposition by the composite position determining unit 53.

In FIG. 5, the frame Fa indicates a frame captured at the K-th (K beingan integer larger than one) time of capturing during panoramic imagecapturing. The frame Fb indicates a frame captured at the (K+1)th timeof capturing during the same panoramic image capturing. That is, theframe Fb is acquired immediately after the frame Fa has been acquired.

As shown in FIG. 5A, the composite position determining unit 53 detectsoverlapping areas (area portions including the same subject) within theframes Fa and Fb, and specifies the overlapping areas as search ranges.

Here, the method of detecting the overlapping areas (area portionsincluding the same subject) within the frames Fa and Fb is not limited,and an arbitrary method can be employed such as one that compares theimage data of frames Fa and Fb by way of image processing.

However, in the present embodiment, as described above, the image dataof one frame is captured each time the digital camera 1 has moved aconstant amount (each time the cumulative value of the angulardisplacement reaches a constant value). Therefore, it is possible toestimate the overlapping areas to some extent based on the constantamount (the constant value of angular displacement). Therefore, in thepresent embodiment, a method is employed such that the overlapping areasare detected as the area portions estimated based on the constant amount(the constant value of angular displacement).

Next, the composite position determining unit 53 defines as references,for example, the left ends of the respective overlapping areas of theframes Fa and Fb, specified as search ranges, and calculates a degree ofsimilarity between candidate areas of the frames Fa and Fb rightwardfrom the reference, one after another. The candidate areas of the framesFa and Fb whose degree of similarity is to be calculated areequidistantly spaced from the reference.

The positions and number of candidate areas whose degree of similarityis to be calculated, are not limited. However, for ease of description,it is herein assumed that four candidate areas have been selected atpredetermined equally-spaced intervals.

In FIG. 5A, four candidate areas a1 to a4 are selected from the frameFa, and four candidate areas b1 to b4 are selected from the frame Fb.The similarities are calculated respectively between the candidate areasa1 and b1, between the candidate areas a2 and b2, between the candidateareas a3 and b3, and between the candidate areas a4 and b4.

Although there is no particular limitation to the method of calculatinga degree of similarity, in the present embodiment, a method is employedthat calculates a degree of similarity using the sum of squareddifferences, i.e., SSD (Sum of Squared Differences), in luminancebetween respective pixels of two candidate areas to be compared.

Here, as shown in FIG. 5B, it is assumed that the highest degree ofsimilarity (the least SSD) has been calculated between the candidateareas a2 and b2, for example.

In this case, as shown in FIG. 5C, the image data of a partial area Fapof the frame Fa and the image data of a partial area Fbp of the frame Fbare combined so that the candidate area a2 of the frame Fa and thecandidate area b2 of the frame Fb are stuck together as respective edgesof the partial areas.

This means that the candidate area a2 becomes an optimal composite areafor the frame Fa, and the candidate area b2 becomes an optimal compositearea for the frame Fb.

In the above, a description has been given of the functionalconfiguration of the digital camera 1 according to the present inventionwith reference to FIGS. 2 to 5.

In the following, a description will be given of the image captureprocessing carried out by the digital camera 1 having such a functionalconfiguration with reference to FIG. 6.

FIG. 6 is a flowchart showing one example of flow of the image captureprocessing.

In the present embodiment, the image capture processing starts when thepower supply (not shown) of the digital camera 1 is turned on.

In step S1, the image capture control unit 51 shown in FIG. 2 executesoperation detection processing and initialization processing.

The operation detection processing is intended to mean a process ofdetecting the states of switches of the operation unit 20. By executingthe operation detection processing, the image capture control unit 51can detect whether the normal image capture mode or the panoramic imagecapture mode is specified as the operation mode.

The initialization processing, in the present embodiment, includes aprocess of setting a constant value of the angular displacement and athreshold value (e.g., 360 degrees) as an upper limit of the angulardisplacement.

More specifically, the constant value of the angular displacement andthe threshold value (e.g., 360 degrees) as an upper limit of the angulardisplacement are stored in advance in the ROM 12 shown in FIG. 1, andthose values are set after being read from the ROM 12 and written intothe RAM 13. The constant value of the angular displacement is used inthe determination process of step S35 of FIG. 7, which will be describedlater. On the other hand, the threshold value (e.g., 360 degrees) as anupper limit of the angular displacement is used in the determinationprocess of step S44 of FIG. 7.

Furthermore, in the present embodiment, the angular displacementdetected by the angular velocity sensor 22 is accumulated, for example,as shown in steps S34 and S39 of FIG. 7, which will be described later.As a result of accumulation thereof, a cumulative angular displacementand a total angular displacement are stored in the RAM 13. Thedifference between the cumulative angular displacement and the totalangular displacement will be described later. Therefore, processes ofresetting the cumulative angular displacement and the total angulardisplacement to 0 are included in the initialization processing in thepresent embodiment. The cumulative angular displacement is compared withthe above-described constant value in the determination process of stepS35 of FIG. 7, which will be described later. On the other hand, thetotal angular displacement is compared with the above-describedthreshold value in the determination process of step S44 of FIG. 7,which will be described later.

Furthermore, the initialization processing in the present embodimentincludes a process of resetting an error flag to 0. The error flag isintended to mean a flag that is set to 1 when an error has occurredduring the panoramic image capture processing (see step S43 of FIG. 7,which will be described later).

In step S2, the image capture control unit 51 starts live-view imagecapture processing and live-view image display processing.

This means that the image capture control unit 51 controls the imagecapturing unit 16 and the image processing unit 17 to continue the imagecapturing by the image capturing unit 16. While the image capturing iscontinued by the image capturing unit 16, the image capture control unit51 temporarily stores in a memory (the storing unit 18, in the presentembodiment) image data of frames sequentially outputted from the imageprocessing unit 17 via the image capturing unit 16. Such a series ofcontrol processing by the image capture control unit 51 is what isreferred to as “live-view image capture processing”.

Also, the image capture control unit 51 controls a display control unit(not shown) to sequentially read the image data of each frametemporarily stored in the memory (the storing unit 18, in the presentembodiment) at the time of the live-view image capturing and to causethe display unit 19 to sequentially display each frame corresponding tothe image data. Such a series of control processing by the image capturecontrol unit 51 is what is referred to as “live-view image displayprocessing”. Hereinafter, a frame that is displayed on the display unit19 by the live-view image display processing is referred to as“live-view image”.

In step S3, the image capture control unit 51 determines whether or notthe shutter switch 41 is half pressed.

Here, “half press” is intended to mean an operation of pressing theshutter switch 41 of the operation unit 20 half way down (as far as apredetermined position but short of its lower limit). Hereinafter, suchan operation is also referred to as “half press operation” asappropriate.

If the shutter switch 41 is not half pressed, a determination of NO ismade in step S3, and control proceeds to step S12.

In step S12, the image capture control unit 51 determines whether or notan instruction has been given to terminate the processing.

In the present embodiment, it is assumed that, with regard to theinstruction to terminate the processing, notification is used whichindicates that the power supply (not shown) of the digital camera 1 hasbeen turned off, but there is no particular limitation thereto.

Therefore, in the present embodiment, when the image capture controlunit 51 is notified that the power supply has been turned off, adetermination of YES is made in step S12, and the entire image captureprocessing ends.

On the other hand, when the power supply is on, since the image capturecontrol unit 51 is not notified that the power supply has been turnedoff, a determination of NO is made in step S12, control goes back tostep S2, and the processes thereafter are repeated. This means that, inthe present embodiment, as long as the power is on, the loop processingfrom steps S3: NO, to S12: NO, is repeated until the shutter switch 41is half pressed, and the image capture processing enters into a waitingstate.

When the shutter switch 41 is half pressed during the live-view imagedisplay processing, a determination of YES is made in step S3, andcontrol proceeds to step S4.

In step S4, the image capture control unit 51 controls the imagecapturing unit 16 to execute what is called AF (Auto Focus) processing.

In step S5, the image capture control unit 51 determines whether or notthe shutter switch 41 has been full pressed.

If the shutter switch 41 has not been full pressed, a determination ofNO is made in step S5. In such a case, control goes back to step S4, andthe processes thereafter are repeated. This means that, in the presentembodiment, the loop processing of step S4 and step S5: NO, is repeateduntil the shutter switch 41 is full pressed, and the AF processing isexecuted each time the loop processing is repeated.

After that, when the shutter switch 41 is full pressed, a determinationof YES is made in step S5, and control proceeds to step S6.

In step S6, the image capture control unit 51 determines whether or notthe operation mode currently set is the panoramic image capture mode.

If the operation mode is not the panoramic image capture mode, i.e., thenormal image capture mode is currently set, a determination of NO ismade in step S6, and control proceeds to step S7.

In step S7, the image capture control unit 51 executes theabove-described normal image capture processing.

This means that image data of one frame outputted from the imageprocessing unit 17 immediately after the full press operation is storedin the removable media 31, as a recording target. With this, the normalimage capture processing of step S7 ends, and control proceeds to stepS12. Since the processes of steps S12 and thereafter have been alreadydescribed above, a description thereof is omitted here.

On the other hand, if the panoramic image capture mode is currently set,a determination of YES is made in step S6, and control proceeds to stepS8.

In step S8, the image capture control unit 51 executes the panoramicimage capture processing described above.

A detailed description of the panoramic image capture processing will begiven later with reference to FIG. 7, but basically, image data of apanoramic image is generated and stored in the removable media 31 as arecording target. With this, the panoramic image capture processing ofstep S8 ends, and control proceeds to step S9.

In step S9, the image capture control unit 51 determines whether or notthe error flag is set to 1.

A detailed description will be given later with reference to FIG. 7, butif the image data of the panoramic image is stored in the removablemedia 31 as a recording target and thereby the panoramic image captureprocessing of step S8 ends properly, the error flag is set to 0. In sucha case, a determination of NO is made in step S9, and control proceedsto step S12. Since the processes of steps S12 and thereafter have beenalready described above, a description thereof is omitted here.

On the other hand, if some error has occurred during the panoramic imagecapture processing of step S8, the panoramic image capture processingends improperly. In such a case, since the error flag is set to 1, adetermination of YES is made in step S9, and control proceeds to stepS10.

In step S10, the image capture control unit 51 displays the errorcontent on the display unit 19. Specific examples of the error contentto be displayed will be described later.

In step S11, the image capture control unit 51 releases the panoramicimage capture mode and resets the error flag to 0.

After that, control goes back to step S1, and processes thereafter arerepeated. This means that the image capture control unit 51 waits foranother image capture operation by the user.

In the above, a description has been given of flow of the image captureprocessing with reference to FIG. 6.

In the following, a description will be given of a detailed flow of thepanoramic image capture processing of step S8 from the image captureprocessing of FIG. 6 with reference to FIG. 7.

FIG. 7 is a flowchart showing a detailed flow of the panoramic imagecapture processing.

As described above, when the shutter switch 41 is fully pressed in thepanoramic image capture mode, determinations of YES are made in steps S5and S6 of FIG. 6, control proceeds to step S8, and the followingprocessing is executed as the panoramic image capture processing.

This means that, in step S31 of FIG. 7, the image capture control unit51 acquires angular displacement from the angular velocity sensor 22.

In step S32, the image capture control unit 51 determines whether or notthe angular displacement acquired in the process of step S31 is greaterthan 0.

If the user has not moved the digital camera 1, the angular displacementis equal to 0. Therefore, a determination of NO is made in step S32, andcontrol proceeds to step S33.

In step S33, the image capture control unit 51 determines whether or nota predetermined time period for which the angular displacement continuesto be 0 has elapsed. As the predetermined time period, for example, atime period can be employed that is appropriately longer than a timeperiod necessary for the user to start to move the digital camera 1after the full press of the shutter switch 41.

If the predetermined time period has not yet elapsed, a determination ofNO is made in step S33, control goes back to step S31, and the processesthereafter are repeated. This means that, as long as the duration of astate in which the user does not move the digital camera 1 does notexceed the predetermined time period, the image capture control unit 51repeats the loop processing from steps S31 to S33: NO, and thereby thepanoramic image capture processing enters into a waiting state.

During such a waiting state, if the user moves the digital camera 1, theangular displacement acquired from the angular velocity sensor 22becomes greater than 0. In such a case, a determination of YES is madein step S32, and control proceeds to step S34.

In step S34, the image capture control unit 51 updates the cumulativeangular displacement by adding the angular displacement acquired in theprocess of step S31 to the previous cumulative angular displacement(cumulative angular displacement=previous cumulative angulardisplacement+angular displacement). In this way, the value stored in theRAM 13 as the cumulative angular displacement is updated.

The cumulative angular displacement is intended to mean an accumulatedvalue of the angular displacement and indicates the moving amount of thedigital camera 1.

Here, in the present embodiment, each time the user moves the digitalcamera 1 by a constant amount, it is assumed that image data of oneframe (target to be combined) for generation of image data of apanoramic image in progress, which will be described later, is suppliedfrom the image processing unit 17 to the image composition unit 52.

For this purpose, a cumulative angular displacement corresponding to the“constant amount” as a moving amount of the digital camera 1 has beengiven in advance as the “constant value” in the initializationprocessing of step S1 of FIG. 6.

In the present embodiment, each time the cumulative angular displacementreaches the constant value, image data of one frame (target to becombined) is supplied from the image processing unit 17 to the imagecomposition unit 52, and the cumulative angular displacement is reset to0.

Such a series of processing is carried out as processes of a subsequentstep S35 and thereafter.

In step S35, the image capture control unit 51 determines whether or notthe cumulative angular displacement has reached the constant value.

If the cumulative angular displacement has not yet reached the constantvalue, a determination of NO is made in step S35, control goes back tostep S31, and processes thereafter are repeated. This means that, unlessthe cumulative angular displacement reaches the constant value due tothe fact that the user has moved the digital camera 1 by the constantamount, the image capture control unit 51 repeats the loop processingfrom steps S31 to S35.

After that, when the cumulative angular displacement has reached theconstant value due to the fact that the user has moved the digitalcamera 1 by the constant amount, a determination of YES is made in stepS35, and control proceeds to step S36.

In step S36, the image composition unit 52 acquires image data of oneframe from the image processing unit 17 under control of the imagecapture control unit 51.

This means that, after control proceeds to step S36 due to the fact thatthe cumulative angular displacement has reached the constant value, theimage capture control unit 51 issues an acquisition instruction to theimage composition unit 52.

Upon receiving the acquisition instruction, the image composition unit52 acquires image data of one frame from the image processing unit 17,as the process of step S36.

In step S37, the composite position determining unit 53 calculates adegree of similarity between the image data of the frame acquired in theprocess of step S36 and the image data of the previously acquired framein units of candidate areas.

In the above-mentioned example of FIG. 5, the frame currently acquiredin the process of step S36 corresponds to the frame Fb, and the framepreviously acquired therein corresponds to the frame Fa. The degree ofsimilarity between each of the candidate areas b1 to b4 of the frame Fband each of the candidate areas a1 to a4 of the frame Fa is calculated.

In step S38, the image composition unit 52 defines as composite areas apair of candidate areas of the highest degree of similarity from amongthe pairs of candidate areas, the degree of similarity between which iscalculated in the process of step S37, and thereby generates data of apanoramic image in progress.

Here, the panoramic image in progress is intended to mean an imageshowing an area captured up to now from among the panoramic image to begenerated when a full press operation is performed in a state in whichthe panoramic image capture mode is selected.

When the process of step S38 is executed for the K-th (K being aninteger larger than 1) time, the image data of the K-th frame has beenalready acquired in the K-th process of the preceding step S36.Therefore, the K-th (current) panoramic image in progress is an imagecomposed by sticking together partial areas of respective first to(k−1)th frames (i.e., a panoramic image in progress at the (k−1)th time)and the K-th frame with respective right and left composite areas, theformer corresponding to the candidate area a2 in the example of FIG. 5,and the latter corresponding to the candidate area b2 in the example ofFIG. 5.

In step S39, the image capture control unit 51 updates the total angulardisplacement by adding the current cumulative angular displacement,which is approximately equal to the constant value, to the previoustotal angular displacement (total angular displacement=previous totalangular displacement+cumulative angular displacement). In this way, thevalue stored in the RAM 13 as the total angular displacement is updated.

In step S40, the image capture control unit 51 resets the cumulativeangular displacement to 0. This means that the value stored in the RAM13 as the cumulative angular displacement is updated to 0.

In this way, the cumulative angular displacement is used for controllingthe timing of the image data of one frame (target to be combined) beingsupplied from the image processing unit 17 to the image composition unit52, i.e., the timing of issuing the acquisition instruction. For thispurpose, the cumulative angular displacement is reset to 0 each time theconstant value is reached and the acquisition instruction is issued.

Accordingly, even if the cumulative angular displacement is used, theimage capture control unit 51 cannot recognize up to which position thedigital camera 1 has moved from the panoramic image capture processingstarting up until the present.

In order to make it possible for the image capture control unit 51 torecognize up to which position the digital camera 1 has moved, in thepresent embodiment, the total angular displacement is employed inaddition to the cumulative angular displacement.

The total angular displacement is an accumulated value of the angulardisplacement, but is not reset to 0 even if the cumulative angulardisplacement has reached the constant value, and is always accumulateduntil the panoramic image capture processing ends (more precisely, untilthe process of step S46, which will be described later, is executed).

After the total angular displacement is updated in the process of stepS39 and the cumulative angular displacement is reset to 0 in the processof step S40, control proceeds to step S41.

In step S41, the image capture control unit 51 determines whether or nota release operation has been performed.

If no release operation has been performed, i.e., if the shutter switch41 is still fully pressed by the user, a determination of NO is made instep S41, and control proceeds to step S42.

In step S42, the image capture control unit 51 determines whether or notany error has occurred in image capturing.

While there is no particular limitation with regard to error in imagecapturing, in the present embodiment, as an error in image capturing,any movement of the digital camera 1 in an oblique, upward, downward, orreverse direction by more than a predetermined value is employed.

If no error in image capturing has occurred, a determination of NO ismade in step S42, and control proceeds to step S44.

In step S44, the image capture control unit 51 determines whether or notthe total angular displacement has exceeded the threshold value.

As described above, the total angular displacement is intended to meanan accumulated value of angular displacement from the point in time whenthe panoramic image capture processing starts (when full press operationhas been performed) until the point in time when the process of step S44is executed.

In the present embodiment, the maximum possible amount by which the usercan move the digital camera 1 during the panoramic image capturing ispredetermined. The total angular displacement corresponding to the“maximum moving amount” as a moving amount of the digital camera 1 isgiven in advance as the “threshold value” in the initializationprocessing of step S1 of FIG. 6.

This means that, in the present embodiment, the fact that the totalangular displacement has reached the threshold value means that thedigital camera 1 has moved by the maximum moving amount.

Therefore, if the total angular displacement has not reached thethreshold value, i.e., the moving amount of the digital camera 1 has notreached the maximum moving amount, the user can still continue to movethe digital camera 1. In this case, a determination of NO is made instep S44, control goes back to step S31, and processes thereafter arerepeated.

Assuming that a state in which the time period for which the angulardisplacement continues to be zero reaches a predetermined time period(i.e., the digital camera 1 does not move for a predetermined timeperiod) is included as one error in image capturing, as long as the fullpress operation continues in a state in which no error has occurred, theloop processing from steps S31 to S44: NO is repeated.

After that, if a release operation is performed in a state in which noerror has occurred (i.e., YES is determined in the process of step S41)or if the digital camera 1 has moved by more than the maximum movingamount (i.e., YES is determined in the process of step S44), thencontrol proceeds to step S45.

In step S45, the image capture control unit 51 generates image data of apanoramic image by the image composition unit 52, and stores it in theremovable media 31 as the image data to be recorded.

In the present embodiment, since image data of a panoramic image inprogress is generated each time image data of a frame is acquired, theimage data of the panoramic image in progress that has been generated atthe time of the process of step S45 is employed as the eventual imagedata of the panoramic image.

In step S46, the image capture control unit 51 resets the total angulardisplacement to 0.

With this, the panoramic image capture processing ends properly. Thismeans that the process of step S8 of FIG. 6 ends properly, and adetermination of NO is made in the process of the next step S9. Sincethe processes after NO is determined in the process of step S9 have beenalready described above, a description thereof is omitted here.

During the series of processes described above, if some error occurs,i.e., if YES is determined in the process of step S33, or if YES isdetermined in the process of step S42, then control proceeds to stepS43.

In step S43, the image capture control unit 51 sets the error flag to 1.

In this case, the process of step S45 is not executed, i.e., no imagedata of any panoramic image is recorded, and the panoramic image captureprocessing ends improperly.

This means that the process of step S8 of FIG. 6 ends improperly, YES isdetermined in the process of the next step S9, and error content isdisplayed in the process of step S10.

What is displayed as error content in this case is not limited asdescribed above, and, for example, a message such as “Image capturefailure” or “Time is over” may be displayed.

As described above, the digital camera 1 of the present embodiment isprovided with an image capturing unit 16, which sequentially outputsdata of each image sequentially acquired by capturing an image at apredetermined time interval, as image data of a frame.

The digital camera 1 of the present embodiment is also provided with animage composition unit 52 and a composite position determining unit 53.

The image composition unit 52 acquires image data of a frame outputtedfrom the image capturing unit 16 by the image processing unit 17 eachtime the digital camera 1 moves by a constant amount (each time theabove-described cumulative angular displacement reaches a constantvalue) and stores it in the storing unit 18. Furthermore, the imagecomposition unit 52 combines at least a part of image data of aplurality of frames that have been cumulatively stored in the storingunit 18, and thereby sequentially generates image data of a panoramicimage in progress (composite image).

The composite position determining unit 53 calculates a degree ofsimilarity for a plurality of pairs of candidate areas between pairedcandidate areas within respective partial areas of two adjacent frames,from among the targets to be combined by the image composition unit 52,and determines the composite positions of the two adjacent frames basedon the pair of candidate areas of the highest degree of similarity.

In this way, as described with reference to FIG. 5, parts of a pluralityof frames are combined at proper positions. This will prevent image dataof frames from being combined into image date of a composite image inwhich the same subject is observed in double, or a subject supposed tobe present is missing in the vicinity of the composite position of twoframes. As a result, it is possible to acquire image data of a panoramicimage that will not feel strange.

It should be noted that the present invention is not limited to theembodiment described above, and modifications and improvements theretowithin a scope in which an object of the present invention can berealized, are included in the invention.

For example, it has been described in the embodiment described abovethat when a composite position is determined by the composite positiondetermining unit 53, candidate areas of the same length as the number ofpixels in a vertical direction of a frame have been employed, and adegree of similarity has been determined between the entire candidateareas.

However, it is not necessary to compare the entire candidate areas, anda degree of similarity may be determined between parts of respectivecandidate areas.

Furthermore, for example, in the embodiment described above, with regardto the candidate areas to be used by the composite position determiningunit 53 to determine the composite positions, length, width, positions,and number thereof have been fixed.

However, at least a part of the length, width, positions, and number ofcandidate areas may be made variable. This means that the compositeposition determining unit 53 may be provided with a function of variablysetting at least a part of the length, width, positions, and number ofcandidate areas, the degree of similarity between which is to becalculated.

Furthermore, although a description was given in the embodimentdescribed above that the angular displacement of the digital camera 1 isdetected by way of the angular velocity sensor 22, the method ofdetecting the angular displacement is not limited to this.

For example, a method may be employed such that the angular displacementof the digital camera 1 is detected by way of image processing byanalyzing the live-view image.

Furthermore, although a description was given in the embodimentdescribed above that the forms of the panoramic image in progress andthe panoramic image are landscape-oriented, the present invention is notlimited to this. The forms of the panoramic image in progress and thepanoramic image may be portrait-oriented, i.e., elongated in a directionin which the digital camera 1 is moved. Also, the image generated by theimage composition unit 52 is not limited to a panoramic image, and theimage composition unit 52 may generate any kind of a wide angle image bycombining a plurality of frames, as long as the wide angle image thusgenerated has a wider angle of view than that of one of the frames.

Furthermore, a description was given in the embodiment described abovethat the image processing apparatus according to the present inventionis configured by a digital camera.

However, the present invention is not limited to this and can be appliedto any electronic device provided with a function capable of generatinga panoramic image. For example, the present invention can be applied toa portable personal computer, a portable navigation device, a portablegame device, and the like.

The series of processing described above can be executed by hardware andalso can be executed by software.

In a case in which the series of processing is to be executed bysoftware, a program configuring the software is installed from a networkor a storage medium into an image processing apparatus or a computerthat controls the image processing apparatus. The computer may be acomputer embedded in dedicated hardware. Alternatively, the computer maybe a computer capable of executing various functions by installingvarious programs, i.e., a general-purpose personal computer, forexample.

The storage medium containing the program can be configured not only bythe removable media 31 distributed separately from the device main bodyfor supplying the program to a user, but also by a storage medium or thelike supplied to the user in a state incorporated in the device mainbody in advance. The removable media 31 is composed of a magnetic disk(including a floppy disk), an optical disk, a magnetic optical disk, orthe like, for example. The storage medium supplied to the user in thestate incorporated in the device main body in advance includes the ROM12 storing the program, a hard disk included in the storing unit 18, andthe like, for example.

It should be noted that, in the present description, the step describingthe program stored in the storage medium includes not only theprocessing executed in a time series following this order, but alsoincludes processing executed in parallel or individually, which is notnecessarily executed in a time series.

1. An image processing apparatus, comprising: an acquiring unit thatacquires a plurality of images which have been captured continuously; asetting unit that sets candidate areas for determining compositepositions in adjacent images, from the plurality of images acquired bythe acquiring unit; a composite position determining unit thatdetermines composite positions of the adjacent images based on a degreeof similarity between the candidate areas set by the setting unit; and acomposite image generating unit that combines the adjacent images at thecomposite positions determined by the composite position determiningunit.
 2. An image processing apparatus as set forth in claim 1, whereinthe setting unit sets a plurality of candidate areas inside areascorresponding to the adjacent images, and the composite positiondetermining unit determines, as the composite positions of the adjacentimages, candidate areas of the adjacent images of the highest degree ofsimilarity, from among the candidate areas thus set.
 3. An imageprocessing apparatus as set forth in claim 1, wherein the setting unitsets at least some among length, width, position, and number ofcandidate areas, for which a degree of similarity is to be calculated.4. An image processing apparatus as set forth in claim 1, comprising animage capturing unit, wherein the acquiring unit acquires a plurality ofimages, each having a first angle of view, by sequentially performingimage capturing by the image capturing unit at predetermined timeintervals.
 5. An image processing apparatus as set forth in claim 1,wherein the candidate areas are sets of pixels constituting a line or arectangle.
 6. An image processing method of an image processingapparatus having an acquiring unit that acquires a plurality of imageswhich have been captured continuously, the method comprising: a settingstep of setting candidate areas for determining composite positions inadjacent images, from the plurality of images acquired by the acquiringunit; a composite position determining step of determining compositepositions of the adjacent images based on a degree of similarity betweenthe candidate areas set in the setting step; and a composite imagegenerating step of combining the adjacent images at the compositepositions determined by the composite position determining step.
 7. Astorage medium readable by a computer that controls an image processingapparatus having an acquiring unit that acquires a plurality of imageswhich have been captured continuously, the storage medium having storedtherein a program causing the computer to function as: a setting unitthat sets candidate areas for determining composite positions inadjacent images, from the plurality of images acquired by the acquiringunit; a composite position determining unit that determines compositepositions of the adjacent images based on a degree of similarity betweenthe candidate areas set by the setting unit; and a composite imagegenerating unit that combines the adjacent images at the compositepositions determined by the composite position determining unit.